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CHAPTER 1/P. S.E.N. REMEDIAL 
READING AND MATH 
1986-87 



0 The 1986-87 Chapter 1 program was satisfactorily 

implemented. Teachers and administrators overwhelmingly 
approved of the services that were provided. 

0 Eligible English-speaking public and nonpublic school 
students did not meet the program objective in reading. 
Nevertheless, program students showed larger gains in 
reading performance this year than last year. 

0 Limited English-proficient (LEP) students met one part of 
the dual-criterion program objective in reading. 

0 Eligible students met and exceeded the program objectives 
in math. 

The Division of Special Education Chapter 1/P. S.E.N. 
Remedial Reading and Mathematics Program was designed to provide 
remedial instruction in reading and/or math to eligible students 
in self-contained special education classes in public and 
nonpublic schools that supplemented their regular instruction. A 
secondary goal was to provide pre- and inservice training to 
special education teachers. The program served a total of 4,910 
students. Four thousand three hundred and fifteen received 
instruction in reading; 44 of these received reading instruction 
in Spanish. Another 416 received instruction in both reading and 
math, and 179 received only math instruction. 

To measure academic progress, students received remedial 
instruction in reading were pre- and posttested with the 
Metropolitan Achievement Test (M.A.T.). LEP students 
participated in the Leamos Spanish Development Reading Program 

(Leamos) which assessed achievement on an ongoing basis. In 
mathematics, the Individualized Criterion Referenced Test 

(I.C.R.T.) in math provided achievement data, also on an ongoing 
basis . 

The Office of Educational Assessment (O.E.A.) conducted an 
evaluation of program implementation and analyzed achievement 
data reported by the program. The evaluation of program 
implementation consisted of observation of the program at 
representative sites, as well as interviews with program staff; 
staff also filled in staff-development assessment forms. 
Analysis of these data indicated that the program was adequately 
implemented. The remediation consisted of individualized 
diagnostic-prescriptive instruction. Reading instruction was 
based on a holistic, meaning-centered approach; mathematics 
instruction emphasized word problems and practical skills. 
Consultants observed a holistic approach to reading instruction 
at 90 percent of the sites visited. For students receiving 



instruction in both reading and mathematics, field consultants 
observed the integration of reading and math instruction through 
the use of word problems and vocabulary study. 

Program staff and special education teachers received pre- 
service orientation in the fall and inservice staff development 
during monthly workshops held in each region, and on-site 
consultations with program staff. They rated their orientation 
and inservice training favorably. Because of a cutback in 
funding in 1986-87, substantially fewer special education 
classroom teachers reported receiving inservice training than in 
1985-86. For future cycles, teachers requested more training in 
the use of new materials, the application of innovative 
instructional techniques, and the utilization of the word 
processor as a teaching aid. They also requested more on-site 
workshops and more individualized training. 

The student achievement objective for reading in English was 
that 75 percent of the students would demonstrate at least a one- 
point normal curve equivalent (N.C.E.) gain in reading 
comprehension from pretest to posttest on the Metropolitan 
Achievement Test (M.A.T.). Of the 3,255 students for whom 
achievement data were complete, 59.9 percent attained this goal. 
The average gain was 3.9 points (S.D.=12.2). Although this was 
an improvement over the previous year's performance, the 
objective was not attained. Excluding the Primary 1 and 
elementary test levels, students at Primary 2, Intermediate, and 
Advanced levels did meet the objective. Students at the Primary 
1 level showed a statistically significant mean N.C.E. loss on 
pre- and posttest scores on the M.A.T., suggesting the M.A.T. was 
not sufficiently sensitive in measuring the performance of lower- 
functioning students. 

A total of 44 students in bilingual classes were taught 
reading in Spanish. The program objectives were that 80 percent 
of these students would master at least two new skills per 20 
sessions attended, and that 30 percent would master at least five 
new skills per 20 sessions attended. Of the total number of 
students, only 54.5 percent mastered at least two skills, while 
34.1 percent mastered at least five skills. Thus, the objective 
was only partially met. 

The program objectives for math were that 80 percent of the 
students would master new math skills at the rate of two per 20 
sessions attended, and 30 percent would master five or more new 
math skills per 20 sessions attended. Both objectives were met 
in that 98 percent mastered two skills per 20 sessions and 56 
percent mastered at least five skills per 20 sessions attended. 

Recommendations for the next program cycle are: 



0 Offer to classroom teachers (i.e., non-Chapter 1) more 



in-depth discussion of effective instructional 
techniques, innovative materials, and the utilization of 
the word processor as a teaching aid. 

0 Replace the M.A.T. in reading with an individualized 
criterion referenced test, if possible, for lower- 
functioning students in reading, i.e., those now 
categorized at the Primary 1 test level. 
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I . INTRODUCTION 



This report documents the Office of Educational Assessment's 
(O.E.A.'s) evaluation of the 1986-87 Chapter 1/P. S.E.N. Remedial 
Reading and Mathematics Program. Administered by the Division of 
Special Education (D.S.E.), the program was designed to provide 
supplementary reading and mathematics instruction to eligible 
English-speaking and limited English-proficient (LEP) special 
education students at 79 public and six nonpublic schools to 
supplement their regular instruction. A secondary goal was to 
provide pre- and inservice training to special education 
teachers . 

INSTRUCTIONAL MODEL 

Reading instruction utilized a holistic, meaning-centered 
approach. It consisted of an integrated process embracing the 
development of listening, speaking, and writing skills, and 
utilizing students' personal experiences. Math instruction 
focused on computation, numeration, practical skills, and the 
integration of reading and math through the use of word problems 
and the study of relevant vocabulary. 

The public school component used a whole-class model of 
instruction. An entire class either traveled as a unit with its 
participating classroom teacher to join the Chapter 1 teacher in 
a program laboratory, or else the program teacher traveled to 
different special education classrooms. The whole-class model 
was designed to retain class integrity, facilitate the integra- 
tion of basic and remedial instruction, and provide training to 
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classroom teachers. 

The six nonpublic schools luect an individual or small-group 
pull-out model in which students received program instruction 
outside their classrooms for a minimum of 90 minutes per week. 

STAFF 

The staff for the Chapter 1 program consisted of one program 
manager, six coordinators, 13 assistant coordinators, and 85 
teachers. The program manager coordinated program activities, 
oversaw budget issues, kept records, served as O.E.A. liaison, 
presented the program to the Parent Advisory Council, and 
supervised the program coordinators. 

The six coordinators managed the day-to-day operation of the 
program. Assistant coordinators had responsibility for providing 
training and assistance *;o both program and participating 
classroom teachers. The 85 program teachers were responsible for 
on-L".te implementation of the program, including student 
assessment, program-related instruction, and recordkeeping. 
A team consisting of a program teacher and a special education 
classroom teacher provided remedial instruction to supplement the 
regular curriculum. 

PARTICIPATING STUDENTS 

The target population consisted of 4,911 eligible students 
in self-contained special education classes ranging it* age from 
seven to 18 years at 85 community, special, and nonpublic 
schools. A total of 4,315 students (87.9 percent of the total) 



received instruction in reading alone; 44 (one percent) of these 
received reading instruction in Spanish; 416 (9.5 percent) 
received instruction in both reading and math, and 179 (4.1 
percent) received only math instruction. 

Public School Population Receiving Reading Instruction in English 

The largest number of program participants (4,131) received 
reading instruction in English in 79 public schools distributed 
throughout the five boroughs. Forty-five percent attended 
elementary schools, 54.2 percent attended intermediate or junior 
high schools, and 0.8 percent attended high schools. Their ages 
ranged from seven to 18 years. 

LEP Public School Population 

Forty-four LEP students received only reading instruction 
in Spanish at two sites, one in Manhattan and one in the Bronx. 
All students attended elementary schools. Their ages ranged from 
eight to 13 years. 

Nonpublic School Population 

Participating nonpublic school students received reading 
instruction only through the D.S.E. Office of Citywide 
Services.* A total of 140 students were served at six non-public 
schools . 



These are centrally-operated programs serving severely 
handicapped students whose numbers are small. 



REPORT FORMAT 

This report is organized as follows: Chapter II describes 
the program methodology; Chapter III presents the findings 
focusing on pupil achievement and program implementation; Chapter 
IV offers conclusions and recommendations based upon the results 
of the evaluation. 
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II. EVALUATION METHODOLOGY 



EVALUATION QUESTIONS 

The evaluation sought to answer questions in the areas of 
program implementation and student achievement. Evaluation 
questions included the following. 

Implementation Questions 

0 What was the level of program implementation? 

0 What was the quality of program implementation? 

0 Did program teachers use the holistic approach to teach 
reading? 

0 Did program teachers stress life skills in their math 
instruction? 

0 What changes have program personnel instituted in the 
present program cycle? 

Outcome Questions 

0 What was the average normal curve equivalent gain 
achieved by students? 

0 What facilitated the program's effectiveness? 

0 What, if anything, detracted from the program's effec- 
tiveness? 

EVALUATION PROCEDURES 
Sample 

To assess program implementation, O.E.A. field consultants 
observed instruction, and interviewed program and school staff at 
15 program sites (18 percent). O.E.A. selected the sample sites 
according to region, district, school level, program service 
categories and program component. 



Consultants observed 35 class periods and interviewed 15 
program teachers (18 percent) , 86 special education classroom 
teachers, 12 program coordinators and assistant coordinators (63 
percent), and 26 school-level administrators (principals, 
assistant principals, and special education site supervisors). 

A total of 4,271 English-speaking public and non-public . 
school students received reading instruction. There were 3,255 
English-speaking public and non-public school students for whom 
reading achievement data were complete (76.2 percent of the 
total). Data were missing from late admits, students attending 
fewer than 20 days, and those dropping out early . Instruction in 
mathematics was given to 596 students. Complete achievement d*ta 
were available for 589. Data were available on all 44 LEP 
students receiving instruction in reading. 

Instrumentation 

To assess program outcomes, O.E.A. collected information on 
student data retrieval forms on which Chapter 1 teachers entered 
program, student, and achievement information. The Metropolitan 
Achievement Test (M.A.T. ) * measured English-language reading 
achievement; the Leamos Diagnostic/Prescriptive Tests (Leamos) 
provided Spanii —language reading achievement data.** The Math 
Individualized C riterion Referenced Test (I.C.R.T.) measured 

*Forms JS and KS, 1978 Survey Edition. San Antonio, Texas: The 
Psychological Corporation. 

** Spanish Developmental Reading Task Force, Los Angeles Unified 
School District (Paul Amidon and Associates), 1976. 
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achievement in mathematics .* 

O.E.A. developed staff development surveys and interview 
schedules to collect information about training and program 
implementation and factors which contributed to, or detracted 
from, the program's success* Consultants administered interview 
schedules to program teachers, special education classroom 
teachers, school administrators, program coordinators, assistant 
coordinators, and documented additional information on classroom 
observation forms. 

Data Collection 

Consultants interviewed and observed school and program 
staff during a four-month period from February to May 1987. 
O.E.A. distributed data retrieval forms to teachers in September 
at the beginning of the program and collected them at the end of 
the program cycle in May. 

Data Analysis 

O.E.A. calculated the number of skills each student 
mastered, and computed the proportion of objectives mastered by 
each student in each curriculum area as measured by the M.A.T., 
Leamos, and I.C.R.T. For pre- and posttest results, O.E.A. 
computed the percent of students showing an N.C.E. gain on the 
M.A.T. O.E.A. used the correlated t-test model to analyze the 
statistical significance of "normal curve equivalent" (N.C.E.) 



*Tulsa, Oklahoma: Educational Progress (Educational Development 
Corporation) , 1980. 

7 

er|c 7 ? 



gains for the performance of English-proficient students in 
reading. 

To document implementation, O.E.A. tallied responses to 
interview questions, and observation descriptions. O.E.A. paid 
particular attention to staff perceptions of factors which either 
enhanced or inhibited program success. 
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III. EVALUATION FINDINGS 



IMPLEMENTATION 

Chapter 1/P. S.E.N, provided supplementary remedial reading 
and mathematics instruction to eligible students in public and 
nonpublic schools. Students received instruction from a 
remediation team consisting of a program teacher and a special 
education classroom teacher. 

Student Instruction 

Scheduling of instruction was the same as in previous 
cycles. Students receiving reading-only services participated in 
four or five 45-minute sessions per week. Students who received 
both reading and math instruction participated in three 45-minute 
sessions per week in each subject. Those students attending non- 
public schools received at least 90 minutes of reading instruc- 
tion per week. 

A summary of sessions scheduled and sessions attended is 
presented in Table 1. Students receiving both reading and math 
instruction had the highest attendance rates, followed closely by 
students receiving reading instruction in Spanish. These 
students had higher attendance rates than students receiving 
reading-only instruction in English. The attendance for the 
English-speaking public school students receiving reading-only 
instruction (the largest group served) was slightly lower (74.3 
percent) than it was in the previous cycle (77.6 percent). 
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TABLE 1 

Chapter 1/P. S.E.N. Sessions Scheduled and Attended 



Instructional Content 


Number of 
Students 


Mean 
Number of 
Sessions 
Scheduled 


Mean 
Percent of 
Sessions 
Attended 


Readincr Only 








English (public schools) 


4,131 


91.2 


74.3 


Engl ish ( nonpublic 

s chools ) 


140 


44.5 


79.9 


S Dan ish ( OllM 1 P cpfinr>l e ^ 
u^uiixoii \^UiJllU pUHUwlp y 


A A 


ill 1 
111 . 1 


82.7 


TOTAL 


4,315 


89. S 


74.6 


Math 








heading and Math 


416 


76.4 


82.9 


Math Only 


179 


85.1 


81.2 


TOTAL 


596 


80.5 


81.9 



The greatest number of sessions were scheduled for 
students receiving reading instruction in Spanish; the 
fewest for English-speaking non-public school students 
receiving reading-only instruction. 

Attendence was lowest for English-speaking public school 
students receiving reading-only instruction. 
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Reading Instruction 

O.E.A. consultants observed that the holistic approach to 
reading instruction was in use at 27 of the 30 sites visited (90 
percent). This is equivalent to the previous cycle when 
approximately 89 percent of the observed sites were employing 
this technique at the time of observation. Observers noted 
features of lessons and instructional strategies consistent with 
the holistic approach, such as discussion of the readings to 
develop comprehension, preparatory vocabulary work, reading in 
related content areas, use of learning games and role playing, 
discussion of current events (via the reading of newspapers and 
magazines) , and students 1 personal experiences relevant to the 
reading assignments. Consultants reported the integration of a 
writing activity with a reading lesson at 19 sites (63 percent). 
In terms of the percentage of locations observed, this is 
identical to the previous cycle. 

At sites where word processors were available for reading 
and writing instruction, program teachers reported that students 
liked using them, especially to write original stories. 
Teachers and consultants felt computers increased the students' 
motivation. Sixty-two of the 86 special education classroom 
teachers surveyed (72 percent) indicated they would like to learn 
more about utilizing the word processor for writing and reading 
instruction . 

Most program teachers of English-speaking students used the 
M.A.T. pretest results for instructional planning and grouping, 

11 
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but reported that it was not accurate for assessing students' 
abilities at all levels. Several teachers and coordinators 
pointed out that the M.A.T. may be especially insensitive as an 
instrument for measuring levels of lower-functioning students, 
i.e., the test is too difficult for them. 

Program teachers delivering reading instruction in Spanish 
reported that Leamos provided an accurate assessment of their 
students' abilities and was a useful indicator for instructional 
planning. This perception differed from last year, when a larger 
sample of program teachers reported that an instrument more 
suitable to the holistic approach would be preferable. 

Teachers and coordinators frequently reported the following 
program strengths: two teachers coordinating instruction; the 
variety and relevance of instructional materials and teaching 
strategies: greater individualization of instruction; the 
application of the holistic approach, especially the 
incorporation of writing, class discussion, and the use of 
students' own experiences. Coordinators indicated that 
additional ti:<e for cooperative planning between Chapter 1 staff 
and classroom teachers would be beneficial for future cycles. 

Math Instruction 

O.E.A. consultants who observed math classes reported that 
teachers integrated reading with math instruction through the use 
of word problems and the study of vocabulary. They observed 
this integration during all class periods. Teachers also 
emphasized practical skills and used concrete materials in 

12 
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almost all classes. Consultants observed teachers using a wide 
variety of math materials , including abstract and representa- 
tional items as well as concrete manipulative. 

Program teachers reported that they used math I.C.R.T. 
results for instructional planning and grouping. They stated 
that they were able to identify accurately their students' 
specific strengths and weaknesses. 

Program and school staff specifically praised the opportu- 
nity to individualize instruction, the variety of materials, the 
integration of reading and math, the emphasis on practical 
skills, and the effective use of concrete manipulative. 

Non-instructional Staff Activities 

Staff Collaboration . O.E.A. assessed staff collaboration 
by examining responses to interview questions and by observing 
classroom lessons. Classroom teachers and Chapter 1 teachers 
worked together in the classroom, teaching the same lesson to 
different groups or teaching different lessons to individual 
students or the class as a whole. Chapter 1 and classroom 
teachers reported favorably on their cooperation. They reported 
they had adequate time for coordination and instructional 
planning. 

Staff Development . Program coordinators and teachers, and 
special education classroom teachers received pre-service 
orientation in the fall and inservice training during monthly 
workshops held in each region. Another type of inservice 
training offered was on-site consultation for classroom teachers 
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with the coordinating staff. 

Fifty-seven of the 86 participating classroom teachers 
interviewed (66 percent) reported that the pre-service 
orientation they received was satisfactory. This is slightly 
fewer than in the previous cycle, when 71 percent reported their 
orientation was satisfactory. Thirty-three classroom teachers 
(38 percent of those interviewed) indicated they had received 
inservice training. Due to the reduction in funds in 1986-87, 
this is a substantial decrease from 1985-86, when 61 percent of 
teachers interviewed indicated that they had received inservice 
training. 

The inservice training covered the following topics: roles 
and responsibilities of program and classroom teachers; methods 
and materials for teaching reading holistically ; specific 
instructional strategies for LEF students; utilizing assessment 
data to develop student objectives; using the word processor for 
writing and reading instruction; new instructional strategies in 
mathematics; and using new materials. 

Coordinators and program teachers rated their inservice 
training favorably. Of the topics covered, program coordinators 
were most interested in the workshops that addressed the use of 
assessment data to develop student objectives. Program teachers 
were especially interested in new instructional strategies for 
LEP students. For future cycles, coordinators and assistant 
coordinators requested more information about less restrictive 
placement and mainstreaming of Chapter 1 students. 

14 

24 



Coordinators and program teachers also requested more training in 
using the word processor as a learning tool. 

Classroom teachers requested, as they had in the previous 
cycle, that Chapter 1 staff offer more on-site workshops and 
provide more individualized training. They too were 
particularly interested in learning more about new instructional 
techniques in math and for LEP students, using the word processor 
for writing and reading instruction, and utilizing new materials. 

Parent Education and Involvement 

Parent workshops were held at all Chapter 1 sites. As 
before, attendance was variable, but generally low. The parent 
programs focused on reviewing activities related to Chapter 1 and 
other reimbursable programs in special education. 

OUTCOMES 

English Reading Achievement 

The program objective for students receiving reading 
instruction was: 

0 By June 30, 1987, 75 percent of the Chapter 1/P. S.E.N, 
target population receiving remedial reading instruction 
will demonstrate a normal curve equivalent (N.C.E.) gain 
in reading comprehension from pretest to posttest, as 
measured by the M.A.T. in reading. 

For the purpose of determining program success, only those 
students who attended a minimum of 20 days were included when 
assessing achievement outcomes. O.E.A. combined M.A.T. reading 
scores from public and non-public school students in the analysis 
as there was no differentiation in the program objective. 

15 
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Achievement data showed that overall, 59,9 percent of the 
students demonstrated an N.C.E* gain in reading comprehension 
(fall to spring administration) , indicating that the program 
objective was not attained (see Table 2), This represents an 
improvement from the previous cycle, when 55,8 percent of 
students demonstrated an N.C.E. gain. 

At all levels except primary 1, more than 50 percent of the 
students tested demonstrated an N.C.E. gain. Excluding the 
primary 1 and elementary test levels, the mean gain for the three 
remaining groups — primary 2, intermediate, and advanced — was ' 
73.3 percent which virtually met the 75 percent objective. These 
data indicate that the reading instruction program met its 
objective at these test levels. 

Within test levels, the mean N.C.E. gain of 3.9 for the 
entire group was statistically significant and represented a 
small effect size of .32, which may be considered only slightly 
educationally meaningful.* (See Table 3). 

The Chapter 1 program has produced a statistically 
significant decrement in mean performance for the past two 
cycles at the Primary 1 test level (-1.6 N.C.E. s in 1986-87; 
-2.6 N.C.E.s in 1985-85). a'his decline in performance and the 
relatively low outcome for the elementary level (3.9) appear to 
indicate that the program as measured by the M.A.T. was not 

*The effect size, developed by Jacob Cohen, is a ratio of the 
mean gain to the standard deviation of the gain. Effect size 
(E.S.) is interpreted to indicate educational meaning* ulr.eao . 
An E.S. of .8 is considered highly meaningful, while one of .2 
is considered only slightly meaningful. 

16 

ERIC PS 



TABLE 2 

Students* N.C.E. Gains on the M.A.T. in Reading, 

by Test Level 



Test Level 


N 


Students Showing a Gain 


Primary 1 


766 


34.5 


Primary 2 


743 


69.3 


Elementary 


951 


58.8 


Intermediate 


636 


78.6 


Advanced 


159 


71.1 


ALL LEVELS 


3,255 


59.9 



0 Overall, 59.9 percent of the students showed an 
N.C.E. gain. 

0 Students tested at the Intermediate level had the highest 
percentage showing an N.C.E. gain. 

0 Students at the Primary 2, Intermediate, and Advanced 
levels met the program objective. 
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Significance and Effect Size of 
N.C.E. Gains cn the M.A.T. an Reading, iy Test Level 







Pretest 


Posttest 


nSaXl 


Standard 






Level 


Number of 










N.C.E* 


Deviation 




Effect 


Students 


Mean 


S.D. 


Mean 


S.D. 


VCUU \Uj5SJ 


of Difference t 


Size 


Primary 1 


766 


35.4 


16.9 


33.8 


18.1 


(-1.6) 


14.3 


(-3.0*) 


(-.11) 


Primary 2 


743 


35.6 


13.0 


41.7 


15.2 


6.1 


10.9 


15.3* 


.56 


E^nentary 


951 


36.9 


13.0 


40.7 


13.6 


3.9 


11.0 


10.9* 


.35 


Intermediate 


636 


36.7 


11.6 


43.9 


12.0 


7.3 


10.2 


18.1* 


.71 


Advanced 


159 


39.6 


14.1 


46.3 


16.0 


6.7 


12.7 


6.7* 


.53 


ALL LEVELS 


3,255 


36.3 


13.9 


40.2 


15.5 


3.9 


12.2 


18.3* 


.32 



*2 < .05 



• Mean differences ranged from a loss of 1.6 N.C.E.s at the Primary 1 level to a gain of 7.3 at 
the Intermediate level. 

0 All mean differences were statistically significant. 

0 N.C.E. gains of students tested at the Primary 2, Intermediate and Advanced levels showed 
moderate effect sizes, suggesting moderate educational meaningfulness. 
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effective at the primary 1 and elementary levels. It might be 
that the skills of those tested at that level were already so 
-low, the program could not have made a measurable difference. 

Intermediate and junior high school students demonstrated 
the highest percentage showing improvement (62.9 and 62*6 
percent). (See Table 4.) These students had the largest mean 
N.C.E. gain (4.3), as well as the most educationally meaningful 
gain indicated by the effect size. (See Table 5.) High school 
students had the lowest percentage showing improvement (29.2 
percent) and the smallest mean N.C.E. gain (0.7 percent). The 
large standard deviations of the mean gains suggest that the 
groups of students tested were very heterogeneous, with some 
students making large gains and others showing N.C.E. losses. 

Reading Results for LEP Students 

The program objective for students receiving reading 

instruction in Spanish was: 

0 By June 30, 1987, 80 percent of the bilingual Chapter 1/ 
P. S.E.N, population receiving remedial reading instruc- 
tion in Spanish will master at least two new skills per 
20 sessions attended, and that 30 percent will master at 
least five new skills per 20 sessions attended as 
measured by ongoing administration of the Leamos . 

For the purpose of determining program success, only those 

students attending a minimum of 20 days were included in the 

computation of mastery data. 

Data analysis indicated that 54.5 percent of the students 

(24) mastered at least two skills in 20 sessions. This part of 

the objective was not met. The second part of the objective, 
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TABLE 4 



Students 1 N.C.E, Gains on the M. A.T. 
in Reading, by School Level 



School 
Level 


Total Number 
of Students 


Percent of 
Students Showing a Gain 


Elementary 


1,294 


56.2 


Intermediate 


1,084 


62.9 


Junior High 


848 


62.6 


High School 


24 


29.2 


ALL LEVELS 


3,250 


59.9 



The intermediate school group had the highest percent of 
students showing an N.C.E. gain. 
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TABLE 5 

Significance and Effect Size of 
N.C.E. Gains cn the M.A.T. in Reading, fcy School Level 



Level 


Number of 
Students 


Pretest 
Mean S.D. 


Posttest 
Mean S.D. 


Mean 
N.C.E. 
Gain 


Standard 
Deviation 


t 


: 

Effect 
Size 


Elementary 


1,294 


33.4 


14.6 


36.8 


15.7 


3.3 


13.3 


9.2* 


.25 


Intermediate 


1,084 


38.7 


13.8 


43.0 


15.6 


4.3 


11.4 


12.5* 


.38 


Junior High 


848 


37.3 


11.7 


41.6 


13.9 


4.3 


11.6 


10.7* 


.37 


High School 


24 


50.7 


10.4 


51,4 


12,4 


0.7 


9.1 


0.4 


.03 


MIi LEVELS 


3,250 • 


36.3 


13.8 


40.3 


15.5 


3.9 


12.2 


18.2* 


.32 



*B < .05 



0 Mean N.C.E. gains ranged from 0.7 at the high school level to 4.3 at the intermediate and 
junior high school levels. 

0 All gains were statistically significant except at the high school level; the associated 
effect sizes were small. 

0 Students attending intermediate and junior high schools showed the most meaningful N.C.E. 
gains. 

0 The large standard deviations of the mean gains suggest that the performance of the student 
groups tested was very heterogeneous. 
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however, was met in that 34.1 percent (15) mastered five or more 
skills per 20 sessions. (See Table 6). These outcomes were the 
reverse of the provious year's findings. In 1985-86, students 
met the objective for mastering at least two skills, but did not 
meet the objective for mastering five or more skills. Overall, 
mastery was much less substantial in 1986-87 when 48 percent of 
the students mastered at least 12 skill objectives, compared to 
1985-86 when 75 percent mastered the same number of skills . (See 
Table 7.) 

Hath Achievement 

The objective for students receiving math instructions was: 

0 By June 30, 1987, 80 percent of the students receiving 
remedial math instruction will master two new math skills 
per 20 sessions attended and 30 percent will master at 
least five new skills per 20 sessions attended as 
measured by ongoing administration of the I.C.R.T. 

An analysis of achievement data indicated that both parts of 

the objective were met. Ninety-eight percent mastered at least 

two new skill objectives per 20 sessions attended, and 56 

percent mastered at least five skill objectives per 20 sessions 

attended. (See Table 8.) The mean number of skill objectives 

mastered was 5.8 ( S.D. * 5.7) per 20 sessions attended- The 

distribution was highly skewed, indicating that, overall, 

students readily mastered the objectives. Over 60 percent of the 

students mastered 13 or more objectives, and over 80 percent 

mastered 9 or more. (See Table 9.) The mean number of total 

skills mastered was 14.7 ( S.D. ■ 6.5). This is a reduction from 

1985-86, when the mean number of skills mastered was 26.9. 
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TABLE 6 



Frequency 




Distribution of 


Mastery Rates of Leamos 


Skills 


Mastery 
Rate 8 


Number of 
Students 


Relative 
Percent 


Cumulative 
Percent 


5 or more 


15 


34.1 


34. i 


4 


5 


11.3 


45.4 


3 


3 


ft 


CO o 

52 . 2 


2 


i 


2.3 


54.5, 


1 


19 


43.2 


97.7 


less than 1 


1 


2.3 


100.0 


TOTAL 


44 


100.0 





•Number of skills mastered per 20 sessions attended. 



About 55 percent of LEP students tested with Leamos 
mastered at least two skills per 20 sessions attended. 
This objective was not met. 

About 34 percent of LEP students mastered at least five 
skills per 20 sessions attended, meeting the program 
objective. 
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TABLE 7 



Frequency Distribution of Total 
Leamos Skills Mastered 
(N = 44) 



Number of 

Objectives Number of Relative Cumulative 

Mastered Students Percent Percent 



21 - 24 


11 


25.0 


25.0 


17 - 20 


5 


11.4 


36.4 


13 - 16 


4 


9.0 


45.4 


9-12 


1 


2.3 


47.7 


1-4 


22 


50,0 


97.7 


0 


1 


2.3 


100.0 



Twenty-five percent of program students mastered 21-24 
objectives. 

The largest proportion of the students (50 percent) 
mastered 1-4 objectives. 
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TABLE 8 

Frequency Distribution of Mastery 
Rates of I.C.R.T. Math Skills 



Mastery 


Number of 


Relative 


Cumulative 


Rate 8 


Students 


Percent 


Percent 


o or more 


o ji 


DO • a 


EC «J 


4 


138 


23.4 


79.6 


3 


67 


11.4 


91.0 


2 


42 


7.1 


98.1 


1 


3 


0.5 


98.6 


less than 1 


8 


1.4 


100.0 


TOTAL 


589 


100.0 




a Number of skills m*.. 


-xed per 20 sessions 


attended. 





Over 98 percent of the students mastered at least two new 
skills per 20 sessions attended. 

Over 56 percent mastered at least five new skills per 20 
sersions attended. 
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TABLE 9 



Frequency 


Distribution of T >tal 


I.C.R.T. Math Skills 


Mastered 


Number of 




Students 




f\ V-\ ^\ j-\ M V* A 

UDj ecuives 
Mastered 


Number 


Relative 
Percent 


Cumulative 
Percent 8 


OK /*\ V* MAMA 

*o or more 


A Q 


• 

o o 

8 . 2 


8 . 2 


01 — Oil 


*7 A 


11 . 7 


19 . 9 




10 J 


25 . 7 


45 . 6 




Q Q 


14.9 


60 . 5 


9-12 


123 


20.6 


81.1 


5-8 


71 


11.9 


93.0 


1-4 


31 


5.2 


98.2 


0 


10 


1.7 


99.9 


TOTAL 


596 


99.9 





8 Numbers do not total 100 percent due to rounding. 



About 60 percent of the students mastered 13 or more 
objectives . 

Less than two percent failed to master any new skills. 
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Math Achievement by Skill Area 

The Math I.C.R.T. is divided into the following six skill 
areas: whole number operations , fractions, geometry, measurement, 
decimals, and special topics. Whole number operations, 
fractions, and decimal and percentage sections measure knowledge 
of mathematical concepts, numeration, arithmetic operations, and 
problem-solving skills. Measurement skill areas assess 
competency in length, size, area, and volume. Geometry includes 
work in spatial concepts, lines, angles, shapes, and solids. 
Problems in special topics involve money, graphs, tables, time, 
and Roman numerals. 

As in the previous cycle, over half (57.6 percent) of the 
skills mastered were in the area of whole number operations. 
Almost 30 percent were in special topics; 10.3 percent in 
measurement, 4.2 percent in geometry, and no skills at all were 
mastered in the areas of fractions and decimals. 
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IV. CONCLUSIONS AND RECOMMENDATIONS 
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The Division of Special Education's Chapter 1/P. S.E.N. 
Individualized Reading and Math Services Program provided 
supplementary remedial raading and math instruction to eligible 
students in public and nonpublic schools. 

The Office of Educational Assessment conducted an evaluation 
of the program's implementation and outcomes. Major 
implementation findings included the following: nearly all 
program teachers (90 percent of those observed) employed a 
holistic approach to teach reading; teachers, integrated reading 
and math instruction by using word problems and emphasizing 
practical skills; teachers effectively utilized a wide variety of 
instructional materials; and program teachers reported that 
staff development continued to be excellent. Teachers were 
especially interested in learning more about specific teaching 
strategies, new materials, and computer-assisted learning. 

Analysis of the implementation data indicated that the 
program was satisfactorily implemented except in training- for 
special education classroom teachers. As a result of a cutback 
in funding in 1986-87, only 38 percent of classroom teachers 
interviewed reported receiving inservice training compared to 61 
percent in 1985-86. About the same proportion of classroom 
teachers reported the orientation training they received was 
satisfactory. 

Some program objectives were met while others were not. Of 
the 3,255 English-speaking public and nonpublic school students 
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in reading, 60 percent showed an N.C.E. gain as measured by the 
M.A.T., falling short of the program objective. While test 
performance at the Primary 1 level was relatively poor, students 
at Primary 2, Elementary, Intermediate, and Advanced test levels 
showed N.C.E. gains. 

Excluding Primary 1 and elementary, performance gains were 
moderately educationally meaningful. As in the previous cycle, 
Primary l test level students showed a statistically significant 
mean N.C.E. loss on pre- to posttest scores on the M.A.T. These 
findings bring into question: (1) the sensitivity of the M.A.T. 
for measuring performance, especially of lower-functioning 
students and (2) the utility of the Chapter 1 program for these 
types of students. The variability of student N.C.E. gains also 
suggests that the M.A.T. may not have been appropriate. For many 
students, a higher or lower level, or a different test might 
have been preferable. 

As in the previous cycle, the program for LEP students met 
only one of its two objectives. The objective that 30 percent 
would master at least five skills per 20 sessions was met. The 
objective that 80 percent would master two skills per 20 sessions 
attended was not met. The objective attained in 1986-87 was the 
one not attained in 1985-86. The 1986-87 program was successful 
for higher-functioning students but failed to achieve its 
objective at relatively lower levels of performance. Given the 
small sample of 44 students, the ability to generalize these 
findings is uncertain. 
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In math, the achievement objective was attained, as it was 
in the previous program cycle. Ninety-eight percent of the 
students receiving instruction mastered at least two new skills, 
and 56 percent mastered at least five new skills * The majority 
of the skills students mastered were in the areas of whole number 
operations and special topics such as noney, graphs, tables, 
time, and Roman numerals. 

The conclusions, based upon the findings of this evalua- 
tion, lead to the following recommendations: 

0 Offer to classroom teachers (i.e., non-Chapter 1) more 
in-depth discussion of effective instructional 
techniques, innovative materials, and the utilization of 
the word processor as a teaching aid. 

0 Replace the M.A.T. in reading with an individualized 
criterion referenced test, if possible, for lower- 
functioning students in reading, i.e., those now 
categorized at the Primary 1 test level. 
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CHAPTER 1/P. S.E.N. REMEDIAL 
READING AND MATH 
1986-87* 

EVALUATION SUMMARY 



0 The 1986-87 Chapter 1 program was satisfactorily 

implemented. Teachers and administrators overwhelmingly 
approved of the services that were provided. 

0 Eligible English-speaking public and nonpublic school 
students did not meet the program objective in reading. 
Nevertheless, program students showed larger gains in 
reading performance this year than last year. 

0 Limited English-proficient (LEP) students met one part of 
the dual-criterion program objective in reading. 

0 Eligible students met and exceeded the program objectives 
in math. 

The Division of Special Education Chapter 1/P. S.E.N. 
Remedial Reading and Mathematics Program was designed to provide 
remedial instruction in reading and/or math to eligible students 
in self-contained special education classes in public and 
nonpublic schools that supplemented their regular instruction. A 
secondary goal was to provide pre- and inservice training to 
special education teachers. The program served a total of 4,910 
students. Four thousand three hundred and fifteen received 
instruction only in reading; 44 of these received reading 
instruction in Spanish. Another 416 received instruction in both 
reading and math, and 179 received only math instruction. 

To measure academic progress, students who received remedial 
instruction in reading were pre- and posttested with the 
Metropolitan Achievement Test (M.A.T.). LEP students 
participated in the Leamos Spanish Development Reading Program 
(Leamos) which assessed achievement on an ongoing basis. In 
mathematics, the Individualized Criterion Referenced Test 
(I.C.R.T.) in math provided achievement data, also on an ongoing 
basis. 



*This summary is based on "A Final Evaluation of the Chapter 1/ 
P. S.E.N. Remedial Reading and Math Program 1986-87" prepared by 
the O.E.A. Special Education Evaluation Unit. 

EVALUATION • R&D • TESTES • DATA ANALYSIS 
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The Office of Educational Assessment (O.E.A.) conducted an 
evaluation of program implementation and analyzed achievement 
data reported by the program. The evaluation of program 
implementation consisted of observation of the program at 
representative sites , as well as interviews with program staff; 
staff also filled in staff -development assessment forms. 
Analysis of these data indicated that the program was adequately 
implemented. The remediation consisted of individualized 
diagnostic-prescriptive instruction. Reading instruction was 
based on a holistic , meaning-centered approach; mathematics 
instruction emphasized word problems and practical skills. 
Consultants observed a holistic approach to reading instruction 
at 90 percent of the sites visited. For students receiving 
instruction in both reading and mathematics , field consultants 
observed the integration of reading and math instruction through 
the use of word problems and vocabulary study. 

Program staff and special education teachers received pre- 
service orientation in the fall and inservice staff development 
during monthly workshops held in each region, and on-site 
consultations with program staff. They rated their orientation - 
and inservice training favorably. Because of a cutback in 
funding in 1986-87, substantially fewer special education 
classroom teachers reported receiving inservice training than in 
1985-86. For future cycles, teachers requested more training in 
the use of new materials, the application of innovative 
instructional techniques, and the utilization of the word 
processor as a teaching aid. They also requested more on-site 
workshops and more individualized training. 

The student achievement objective for reading in English was 
that 75 percent of the students would demonstrate at least a one- 
point normal curve equivalent (N.C.E.) gain in reading 
comprehension from pretest to posttest on the Metropolitan 
Achievement Test (M.A.T.). Of the 3,255 students for whom 
achievement data were complete, 59.9 percent attained this goal. 
The average gain was 3.9 points (S.D.=12.2). Although this was 
an improvement over the previous year's performance, the 
objective was not attained. Excluding the Primary 1 and 
elementary test levels, students at Primary 2, Intermediate, and 
Advanced levels did meet the objective. Students at the Primary 
1 level showed a statistically significant mean N.C.E. loss on 
pre- and posttest scores on the M.A.T., suggesting the M.A.T. was 
not sufficiently sensitive in measuring the performance of lower- 
functioning students. 

A total of 44 students in bilingual classes were taught 
reading in Spanish. The program objectives were that 80 percent 
of these students would master at least two new skills per 20 
sessions attended, and that 30 percent would master at least five 
new skills per 20 sessions attended. Of the total number of 
students, only 54.5 percent mastered at least two skills, while 

2 

ERIC 4 * 



34.1 percent mastered at least five skills. Thus, the objective 
was only partially met. 

The program objectives for math were that 80 percent of the 
students would master new math skills at the rate of two per 20 
sessions attended, and 30 percent would master five or more new 
math skills per 20 sessions attended. Both objectives were met 
in that 98 percent mastered two skills per 20 sessions and 56 
percent mastered at least five skills per 20 sessions attended. 

Recommendations for the next program cycle are: 

0 Offer to classroom teachers (i.e., non-Chapter 1) more 
in-depth discussion of effective instructional 
techniques, innovative materials, and the utilization of 
the word processor as a teaching aid. 

0 Replace the M.A.T. in reading with an individualized 
criterion referenced test, if possible, for lower- 
functioning students in reading, i.e., those now 
categorized at the Primary 1 test level. 



